In Long Term Evolution (LTE) of which standardization has been promoted in 3rd Generation Partnership Project (3GPP) as a 3.9th generation wireless transmission system, for extensive improvement of spectrum efficiency from a third generation wireless transmission system, a Multiple Input Multiple Output (MIMO) technology in which wireless transmission is performed using a plurality of transmitting-receiving antennas has been standardized. Based on a space-multiplexing (SDM) technology which is one of the MIMO technology, enhancing in a throughput can be realized without expanding a frequency bandwidth. In addition, currently, LTE-Advanced (LTE-A) has been proposed as a most-likely candidate of a 4th generation wireless transmission system, and standardization activities thereof have been performed actively. In order to achieve a peak throughput of 1 Gbps in a downlink transmission (base station apparatus to mobile station apparatus) in LTE-A, Single User MIMO (SU-MIMO) in which space-multiplexing up to 8 streams is possible has been investigated. SU-MIMO is a MIMO transmission between a base station apparatus having a plurality of transmission antennas and a single mobile station apparatus having a plurality of reception antennas.
However, there is a limit to the number of reception antennas which can be arranged in a mobile station apparatus. Then, it is considered that adopting Multi-user MIMO (MU-MIMO) in which a plurality of mobile station apparatuses carrying out concurrent access forms a virtual large-scale antenna array and in which transmission signals from a base station apparatus to each of the mobile station apparatuses is made to be space-multiplexed is indispensable to improvement of spectrum efficiency. Although MU-MIMO has already been standardized also in LTE, MU-MIMO adopted in LTE is a system referred to as a beam-forming which carries out multiplication of linear filters in a base station apparatus. In this case, since only the space-multiplexing between users such that space-multiplexed transmission signals between users are orthogonal can be performed, there is a limit to the improvement of spectrum efficiency.
Recently, an MU-MIMO technology where a nonlinear process is performed in a base station apparatus side attracts an attention, and investigation of an MU-MIMO THP technology using a nonlinear interference suppression technology which is referred to as Tomlinson Harashima Precoding (THP) such as in particular described in Non-patent Document 1 is active. MU-MIMO THP is a method where a base station apparatus carries out a modulo (Modulo) calculation and performs transmission after subtracting in advance interferences which the mobile station apparatus receives from a desired signal addressed to each of mobile station apparatuses. Performing the Modulo operation enables the signal after the interference subtraction to be prevented from diverging and an increase of a transmit power to be suppressed. The mobile station apparatus, by performing the Modulo operation again for each reception signal, can detect the desired signal with interferences removed.
MU-MIMO THP can suppress an inter-user interferences (IUI) while suppressing the divergence of the transmit power by the Modulo operation, and therefore, can realize excellent transmission performances. Meanwhile, MU-MIMO THP, by rearranging appropriately an order in which transmission signals addressed to each terminal are calculated (this is referred to as ordering), can achieve an extensive improvement of transmission performances. As an MU-MIMO THP system in which the ordering is performed, a BLAST ZF-THP system has been proposed in Non-patent Document 2.